What is parameter uncertainty?

• How confident you are in your parameter estimates \((\theta, \Omega, \Sigma)\)?
• Typical statistics:
  • Standard errors (SE).
  • Relative standard errors (RSE).
  • Confidence intervals (CI).
• Parameter estimates \((\theta, \Omega, \Sigma)\) are obtained with uncertainty
  • Parametric: Variance-covariance matrix (e.g., NONMEM .cov file).
  • Non-parametric:
    • Log-likelihood profiling (LLP)
    • Non-parametric bootstrap
    • Sampling-importance resampling (SIR)
    • Bayesian posterior distribution

Variability vs. uncertainty

• Both were commonly represented by probability distribution.

• Differ conceptually¹:

  • Variability:

    • Inherent difference in the system.
    • Often cannot be reduced with more data.
    • Typically referring to \(\Omega\) and \(\Sigma\) estimates.

  • Uncertainty:

    • How confident we are in our estimates of the system.
    • Often can be reduced with more data
    • Typically refers to SE, RSE or CI of each parameter, including \(\theta\), \(\Omega\) and \(\Sigma\).

Variability vs. uncertainty

Methods to assess parameter uncertainty

Methods to assess parameter uncertainty

• METHOD=FOCE/SAEM/IMP
  • Distributional assumption
    • $COV: assymptotic covariance matrix from covariance step.
    • Good approximation, may not be obtained even if the model is good ($COV step failed)¹.
    • Less realistic: always symmetrical (est \(\pm\) 1.96*se).
  • No distributional assumptions: Assymmetric, more realistic.
• METHOD=BAYES/NUTS:
  • Directly acquire posterior distribution with no distributional assumption.

Methods to assess parameter uncertainty

Log-likelihood profiling (LLP)¹

LLP advantages and disadvantages

• Advantages:
  • Do not assume symmetric normal distribution
  • Method available in open source platform: PsN
• Disadvantages:
  • Assume chi-square distribution for dOFV, which may only be approximately true for NLME¹.
  • Rarely used in PMx modeling
  • One parameter at a time

LLP implementations in PsN

After NONMEM modeling fitting, LLP can be executed using PsN with command like:

llp mod1.ctl -thetas=1,3 -rplots=2 -clean=0 -min_retries=3

• This will evaluate uncertainty of THETA1 and THETA2 using LLP method with first guess of SE using mod1.lst.
• -rplots=2 generates a few basic plots for output.
• -clean=0 keeps all NONMEM fitting records.
• -min_retries=3 usually needed to make sure each model fitting is successful.
  • If NONMEM fails, rerun the same sub-problem, up to 3 times, with reset initial conditions.”

Non-parametric bootstrap

• Assumption: random difference in observations that leads to uncertainty in parameter estimates.

• To obtain CI

  • Imagine an original dataset ({A,B,C,D,E}) with 5 subjects (i.e., ID in NONMEM).
  • Resample and refit each dataset:
    • Bootstrap Sample 1: ({A,C,C,D,E}): new set of parameter estimates 1
    • Bootstrap Sample 2: ({B,B,D,E,A}): new set of parameter estimates 2
    • Bootstrap Sample 3: ({C,D,E,E,E}): new set of parameter estimates 3
    • …
    • Bootstrap Sample 1000: ({B,D,A,E,E}): new set of parameter estimates 1000
  • Summarize 2.5% and 97.5% percentiles across all new sets of parameter estimates

Non-parametric bootstrap advantages and disadvantages

Non-parametric bootstrap implementations in PsN

After NONMEM modeling fitting, non-parametric bootstrap can be executed using PsN with command like:

bootstrap mod1.ctl -samples=500 -seed=12135 -rplots=2

• This will evaluate uncertainty of all model parameters based on 500 refits of the NONMEM model.
• -rplots=2 generates a few basic plots for output.

Sampling importance resampling (SIR)¹

Where to find a SIR Proposal distribution?

• NONMEM covariance step ($COV)-Most typical.
• Limited number (e.g., N=10) of non-parametric bootstrap.
• Educated guess.
• LLP¹.

SIR diagnostics¹

SIR diagnostics¹

SIR advantages

• Do not assume symmetric normal distribution.
• Obtain uncertainty for all parameters at once.
• Overcome non-parametric bootstrap drawbacks:
  • Faster.
  • No model refit: do not need to consider convergence.
  • Do not need to consider covariate imbalance and small sample size.
  • OK to use in MBMA or in the presence of frequentist prior.
• Well-developed diagnostics.
• Method available in open source platforms:
  • PsN
  • NONMEM

SIR disadvantages

• Still slow, although faster than bootstrap.
• Sensitive to proposal distributions¹.
• May need multiple attempts based on diagnostics.

SIR implementations in PsN

After NONMEM modeling fitting, SIR can be executed using PsN with command like:

sir -samples=2500 -resamples=500 -rplots=2 mod1.ctl

• This will evaluate uncertainty of all model parameters based on 500 resampled parameter sets with a M/m ratio of 5:1 (2500:500).

• -rplots=2 generates dOFV and a few other diagnostics.

• What if the diagnostics is not satisfied?

  • If \(DF_{proposal}<DF_{reference}\), inflate proposal distribution variance.

    • sir -samples=2500 -resamples=500 -theta_inflation=1.5 -omega_inflation=1.5 -sigma_inflation=1.5 -rplots=2 mod1.ctl

  • If \(DF_{final}>DF_{reference}\), increase M/m ratio.

    • sir -samples=5000 -resamples=500 -rplots=2 mod1.ctl

SIR Practical Suggestions

• Start with a small batch of sampling / resampling (e.g., 500/100) and check the diagnostics.
  • optimize the SIR conditions (e.g., proposal distribution variance, sampling to resampling ratio) as needed.
• Perform final SIR using the optimized SIR conditions.

Multiple-iteration SIR¹

Multiple-iteration SIR implementations in PsN

After NONMEM modeling fitting, multiple iteration SIR can be executed using PsN with command like:

sir -samples=2500,2500,2500,1000,1000 -resamples=500,500,500,500,500 -rplots=2 mod1.ctl

• This will evaluate uncertainty of all model parameters based on 500 resampled parameter sets with multiple iteration sampling and resampling with M/m ratio of 5:1 (2500:500), 5:1 (2500:500), 5:1 (2500:500), 2:1 (1000:500), and 2:1 (1000:500).
• -rplots=2 generates dOFV and a few other diagnostics.